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Abstract 


This  project  continues  ongoing  efforts  by  the  authors  to  understand 
transactions  costs  within  DoD  acquisition.  Past  studies  by  the  authors  have  been 
constrained  by  the  data  available.  As  part  of  continuing  effort  to  acquire  more  data 
and  take  advantage  of  first-hand  knowledge  of  the  issue,  this  study  analyzes  results 
from  a  survey  of  US  Air  Force  Program  Managers  undertaken  in  2008  by  the 
National  Research  Council  (NRC,  2009). 

The  theoretical  foundations  of  our  supporting  inquiry  come  from  Transaction 
Cost  Economics  (TCE)  and  Agency  Theory — well-established  fields  of  study.  In 
particular,  we  are  concerned  with  the  complications  and  costs  of  dealing  with 
partners  both  outside  DoD  (TCE)  and  within  (Principal-Agent  Problem). 

The  number  of  oversight  reviews  has  steadily  increased,  with  increasingly 
higher-level  involvement.  Accordingly,  the  resources  and  management  attention 
devoted  to  these  reviews  has  also  increased.  Within  that  context,  the  NRC  study 
attempted  to  assess  program  reviews  with  respect  to  value  added  and  various  costs 
incurred.  Our  analysis  of  the  survey  results  distinguishes  between  technical  and 
programmatic  reviews.  Technical  reviews  are  conducted  by  the  program  manager 
(as  principal)  to  monitor  technical  progress  of  the  system  contractors  (agents). 
Programmatic  reviews  provide  management  oversight  of  the  program  manager  (as 
agent)  by  higher-level  authorities  in  DoD  or  Congress  (principals). 

Our  results  suggest  that  program  managers  found  some  real  value  in  some  of 
their  programmatic  reviews,  despite  the  common  perception  that  reviews  create 
excessive  and  burdensome  levels  of  oversight.  In  addition,  we  found  that  program 
mangers  gave  relatively  less  value  to  technical  reviews,  a  result  some  might  find 
counterintuitive. 

Keywords:  Acquisition,  program  management,  transaction  costs,  principal- 
agent,  technical  reviews,  management  oversight 
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I.  Introduction 


Transaction  cost  economics  (TCE)  emphasizes  transaction  costs 
(coordination  costs  and  motivation  costs)  in  addition  to  traditional  production  costs 
and  provides  a  useful  framework  for  studying  defense  acquisition.  In  this  study,  we 
focus  on  the  costs  of  monitoring  and  oversight  incurred  by  the  program  management 
process  within  the  context  of  TCE  and  the  principal-agent  model  (agency  theory). 

We  use  data  from  a  survey  conducted  by  the  National  Research  Council 
(NRC,  2009)  to  understand  the  costs  and  benefits  of  monitoring  and  oversight  from 
the  program  manager’s  perspective.  In  this  NRC  survey,  program  managers  were 
asked  to  subjectively  assess  the  impact/value  of  different  programmatic  and 
technical  reviews.  We  test  the  assumption  that  program  managers  are  more  likely  to 
view  technical  reviews  as  less  costly/more  beneficial  and  view  programmatic  reviews 
as  more  costly/less  beneficial. 

In  the  first  section  of  this  report,  we  present  the  basic  theoretical  foundations 
for  our  study:  transaction  cost  economics  and  the  principal-agent  model.  We 
discuss  efforts  to  measure  transaction  costs  and  the  application  of  the  principal- 
agent  model  to  the  program  manager’s  role  in  the  acquisition  process.  Next,  we 
discuss  the  program  manager’s  perspective  in  terms  of  program  oversight  and 
program  reviews.  We  conclude  this  section  with  a  summary  of  the  NRC  study.  In 
the  last  section,  we  present  the  results  of  our  analysis  of  the  NRC  survey  data — 
which  yielded  some  counterintuitive  results. 
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II.  Theoretical  Foundations 


A.  Transaction  Cost  Economics 

Conventional  economic  analysis  focuses  on  production  costs  (input  costs, 
competition,  learning  curves,  economies  of  scale  and  scope,  etc.).  Coase  (1937) 
was  the  first  to  ask  why  some  firms  produce  goods  and  services  themselves  at 
higher  production  costs  than  can  be  purchased  in  the  marketplace.  The  answer  is 
that  using  the  market  involves  “transaction  costs,”  and  that  these  costs  can  more 
than  offset  production  cost  advantages  from  outsourcing.  In  making  outsourcing 
decisions,  it  is  important  for  management  to  consider  not  only  the  internal  and 
external  production  costs  of  providing  the  good  or  service,  but  also  the  cost  of 
managing  the  transaction  internally  or  externally.  For  example,  consider  DoD’s 
Defense  Contract  Management  Agency  (DCMA).  This  $1.1  billion  organization  is 
made  up  of  10,500  Civilians  and  600  Military  whose  exclusive  responsibility  is  to 
help  manage  and  coordinate  some  300,000  defense  contracts  valued  at  nearly  $950 
billion. 


Transaction  cost  economics  (TCE)  emphasizes  transaction  costs  (the  cost  of 
carrying  out  a  transaction)  in  addition  to  traditional  production  costs.  Transaction 
costs  typically  encountered  organizations  dealing  with  outside  suppliers  or  sub¬ 
contractors  include  the  costs  of  source  selection,  periodic  competition  and 
renegotiation,  contract  management,  and  measuring  and  monitoring  performance. 
Examples  of  transaction  costs  that  occur  inside  an  organization  include  the  costs  of 
hiring  and  managing  employees  and  selecting  and  controlling  equipment  and 
materials.  TCE  views  organizations  as  a  web  of  contractual  relationships.  Each 
relationship — the  acquisition  of  an  input,  employment  of  a  worker,  the  exchange  of  a 
product  or  service  between  supplier  and  customer — is  a  transaction. 

Transaction  costs  can  be  classified  into  two  categories:  coordination  costs 
and  motivation  costs.  Coordination  Costs  include:  1)  Search  and  Information 
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Costs — to  identify  options  and  acquire  timely,  accurate  and  relevant  information  to 
evaluate  alternatives;  2)  Bargaining  and  Decision  Costs — to  choose  an  alternative 
and  negotiate  and  write  a  contract;  and  3)  Monitoring  and  Enforcement  Costs — to 
make  payments  and  measure,  monitor,  and  evaluate  performance.  Motivation  Costs 
include:  1)  Costs  to  promote  productive  effort  and  incentives  to  encourage 
investment  (better,  faster,  cheaper)  and  2)  Costs  to  deter  unproductive  bargaining 
and  costs  of  opportunistic  behavior  renegotiation). 

Four  key  characteristics  of  transactions  can  make  them  more  costly: 
complexity,  uncertainty,  frequency,  and  asset  specificity.  TCE  suggests  that  an 
understanding  of  the  key  characteristics  of  a  transaction  can  help  decision-makers 
improve  the  design  of  contracts,  organizations,  and  other  governance  structures  that 
reduce  transaction  costs  and  improve  the  gains  from  an  exchange  between  buyers 
and  sellers.  In  other  words,  understanding  transaction  costs  may  help  the  firm 
achieve  results  “faster,  better,  and  cheaper.” 


While  TCE  offers  an  attractive  theoretical  foundation  for  competitive  sourcing 
decisions  in  the  private  sector  (e.g.,  Coase,  1937;  Williamson,  1971;  1979;  Alchian  & 
Demsetz,  1972)  it  has  been  applied  less  often  in  a  government  setting  (e.g..  Pint  & 
Baldwin,  1997;  Williamson,  1999).  Many  goods  and  services  required  for 
government  operations  can  be  provided  through  commercial  markets.  The 
Department  of  Defense  (DoD)  relies  heavily  on  outsourcing,  particularly  when 
producing  weapon  systems.  Franck  &  Melese  (2005)  apply  TCE  to  federal 
outsourcing  and  note  that  transaction  costs  vary  widely  and  depend  in  known  ways 
upon  the  attributes  of  the  outsourcing  action. 


Although  often  used  to  refer  to  major  weapon  system  programs,  the  term 
acquisition  can  cover  all  DoD  purchases  from  the  development  and  procurement  of 
weapon  systems,  to  purchasing  of  services  and  support  for  the  military.  Franck, 
Melese  &  Dillard,  (2006)  extended  the  application  of  TCE  to  defense  procurement. 
They  examine  the  effect  of  “asset  specificity”  on  acquisition  programs.  The  “lock-in” 
effect  achieved  by  contractors  that  invest  in  specific  assets,  while  benefiting  the 
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government  in  the  short  run,  can  haunt  the  government  in  the  long  run.  The  risk  is 
that,  after  winning  a  bidding  competition,  a  contractor  that  invests  in  specific  assets 
might  eventually  become  a  sole  supplier  that  “holds  up”  the  government,  resulting  in 
higher  costs,  schedule  delays,  or  disappointing  performance. 

B.  Measuring  Transaction  Costs 

The  primary  insight  of  TCE  is  that  correctly  forecasting  economic  production 
costs  of  government  purchases  or  acquisitions  is  necessary,  but  not  sufficient.  TCE 
emphasizes  another  set  of  costs — coordination  and  motivation  costs,  such  as  search 
and  information  costs;  decision,  contracting,  and  incentive  costs;  measurement, 
monitoring,  and  enforcement  costs,  etc.  (Melese,  Franck,  Angelis  &  Dillard,  2007).  A 
focus  on  transaction  costs  can  improve  cost  estimation  for  DoD  acquisitions  by  (1) 
helping  to  explain  the  systematic  bias  observed  in  initial  cost  estimates,  and  (2) 
increasing  the  general  explanatory  power  of  cost  estimations.  The  traditional  work 
breakdown  structure  (WBS)  approach  may  overlook  some  important  variables, 
resulting  in  initial  cost  estimates  that  are  (1)  not  accurate  and  (2)  biased  toward 
being  unrealistically  low.  Unlike  the  production-function  approach  of  the  WBS,  the 
TCE  approach  focuses  on  coordination  and  motivation  costs  and  other  key 
components  of  major  weapon  system  acquisitions  (Angelis,  Dillard,  Franck  & 

Melese,  2007). 

While  there  are  several  ways  to  define  and  characterize  transaction  costs, 
actually  measuring  them  can  be  difficult.  Wang  (2003)  discusses  a  variety  of 
empirical  studies  that  attempt  to  measure  transaction  costs.  Some  studies  measure 
transaction  costs  directly  by  measuring  the  economic  value  of  resources  used  in 
locating  trading  partners  and  executing  transactions.  For  example,  for  six  months  in 
the  early  1990s,  Ambassador  Flenry  F.  Cooper  (1993;  as  cited  in  Spring,  2002, 

May),  then  director  of  the  Strategic  Defense  Initiative  Organization,  tracked  the  costs 
to  support  the  attainment  of  one  decision  point  in  a  single  program,  which  was  then 
called  the  Theater  High  Altitude  Area  Defense  (THAAD).  Ambassador  Cooper  found 
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that  it  took  75,000  government  labor  hours,  250,000  contractor  labor  hours,  more 
than  one  ton  of  supporting  documents,  all  at  a  cost  of  $22  million  dollars. 

Unfortunately,  such  costs  are  not  routinely  tracked  by  DoD.  Dillard  (2005) 
noted  that  while  some  representatives  from  program  management  offices  keep  an 
accounting  of  travel  and  labor  costs  associated  with  program  reviews,  the  evidence 
is  mostly  anecdotal.  His  research  suggests  that  a  substantial  amount  of  program 
office  funding  is  expended  on  such  items  as  government  agency  or  support 
contractor  assistance  with  supporting  analyses  and  documentation,  presentation 
materials,  frequent  travel  to  the  Pentagon,  and  other  associated  expenses  in 
preparation  for  high-level  reviews. 

In  earlier  work  (Angelis  et  al.,  2007),  we  attempted  to  directly  measure 
transactions  costs  for  defense  acquisition  programs  using  the  expenditures  of  the 
Program  Management  Office  (PMO)  as  an  approximate  measure  of  the  amount  of 
transaction  costs  present  in  an  acquisition  program.  We  found  that  DoD  does  not 
track  PMO  costs  separately;  therefore,  measuring  transaction  costs  directly  or  by 
proxy  from  the  existing  DoD  data  may  not  be  possible.  As  an  alternative,  it  is 
possible  to  directly  measure  contractor  Systems  Engineering/Program  Management 
(SEPM)  costs  as  an  indication  of  contractor  transaction  costs  (Angelis  et  al.,  2008). 
We  calculated  the  ratio  of  SEPM  to  total  costs  for  two  case  studies  (Javelin  and 
ATACMS)  for  which  ex-ante  indicators  of  transaction  costs  had  been  assessed.  The 
results  are  consistent  in  that  the  program  with  ex-ante  indicators  that  suggested 
higher  transaction  costs  also  had  a  significantly  higher  SEPM  ratio. 


Other  empirical  studies  do  not  attempt  to  measure  transaction  costs  directly. 
Instead,  they  use  a  variety  of  proxies,  such  as  complexity,  uncertainty,  frequency, 
and  asset  specificity.  For  example.  Brown,  Flowe  and  Hamel  (2007)  examine  the 
role  of  complexity  by  measuring  the  difference  in  cost  and  schedule  breaches 
between  System-of-Systems  (SoS)  and  single-system  defense  acquisition 
programs.  SoS  programs  are  defined  as  independent  systems  that  are  integrated 
into  a  larger  system  to  provide  unique  capabilities.  The  identification  of  a  program 
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as  SoS  can  be  seen  as  a  proxy  for  complexity.  We  extended  that  work  (Angelis  et 
al.,  2008)  to  coordination  costs  by  comparing  programs  managed  jointly  by  several 
services  to  programs  managed  by  one  service.  Joint  programs  can  be  seen  as 
proxy  for  coordination  costs.  The  use  of  these  proxies  provides  useful  insights  into 
the  effect  of  transaction  costs. 

C.  Principal-Agent  Model 

Another  way  to  look  at  transactions  is  to  consider  the  problem  that  occurs 
when  cooperating  parties  have  different  goals  and  division  of  labor  (Jensen  & 
Meckling,  1976;  Ross,  1973).  Agency  theory  examines  the  interaction  between  the 
party  that  delegates  or  contracts  for  work  (the  principal)  and  the  party  that  agrees  to 
perform  the  work  (the  agent).  The  focus  of  the  principle-agent  model  is  on 
determining  the  optimal  contract  between  the  two  parties.  There  are  two  main 
problems  that  can  arise  in  agency  relationships:  (1)  the  goals  of  the  principal  and 
agent  are  not  aligned,  and  (2)  the  principal  cannot  easily  verify  what  the  agent  is 
actually  doing.  The  theory  can  suggest  which  contract  type  is  most  effective  under 
different  assumptions. 

In  the  simplest  case,  the  model  assumes  that  principals  and  agents  have 
conflicting  goals  &  objectives,  but  an  easily  measured  outcome.  This  case  also 
assumes  that  the  agent  is  more  risk-adverse  than  the  principal,  since  the  agent  has 
only  one  contract  (with  the  principal),  while  the  principal  can  diversify  his/her 
investments  with  several  contracts.  The  type  of  contract  used  can  depend  on  the 
amount  of  information  available  to  the  principal  (see  Demski  &  Feltham,  1978). 

In  many  cases,  there  is  information  asymmetry  because  the  principal  does 
not  know  exactly  what  the  agent  has  done.  In  this  situation,  the  agent  (who 
accomplishes  the  work)  has  an  information  advantage  over  the  principal.  This  can 
lead  to  two  problems  discussed  in  agency  theory:  adverse  selection  and  moral 
hazard.  Adverse  selection  can  occur  if  agents  misrepresent  their  ability  to 
accomplish  the  work.  This  might  provide  one  explanation  for  why  contractors  often 
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fail  to  deliver  weapon  systems  on  time,  within  budget  and  as  specified.  A  moral 
hazard  can  arise  if  the  agent  chooses  not  to  perform  as  promised  and  the  principal 
has  no  way  of  knowing.  For  example,  a  contractor  may  bill  for  a  service  that  was  not 
actually  rendered  or  bill  for  an  amount  in  excess  of  actual  costs.  The  moral  hazard 
occurs  because  the  principal  is  unable  or  unwilling  to  verify  the  agent’s  effort. 

One  way  the  principal  can  deal  with  lack  of  information  is  to  invest  in  data- 
collection  systems  such  as  budgeting  systems,  cost-accounting  systems  and 
performance-measurement  systems.  The  principal  can  also  establish  reporting 
procedures  such  as  programmatic  and  technical  reviews,  as  well  as  additional  layers 
of  oversight,  as  shown  in  Figure  1  below  for  defense  acquisition  programs. 


Program  Oversight 
and  Execution 


Acquisition  Support 
and  Sustainment 


Figure  1.  Four  Tiers  of  Major  Program  Reporting 

(NRC,  2009) 

Note:  The  meaning  of  each  acronym  is  as  follows:  DAE,  Defense  Acquisition  Executive;  USD 

(AT&L),  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics;  SECAF, 
Secretary  of  the  Air  Force;  CSAF,  Chief  of  Staff  of  the  Air  Force;  MAJCOM  HQ,  Major 
Command  Headquarters;  SAE,  Service  Acquisition  Executive,  SAF/AQ,  Assistant  Secretary 
of  the  Air  Force  for  Acquisition;  PEO,  Program  Executive  Officer;  PM,  Program  Manager 
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In  addition,  the  principal  may  use  a  contract  that  specifies  the  desired 
outcome  (as  opposed  to  effort  or  behavior)  to  align  goals  and  incentives.  Such  an 
arrangement  transfers  the  risk  of  performance  to  the  agent.  For  example,  the 
government  can  use  firm,  fixed-price  contracts  to  acquire  goods  and  services.  Such 
contracts  place  the  responsibility  of  performance  on  the  contractor  or  supplier  and 
reduce  the  cost  and  schedule  risk  to  the  government.  Naturally,  contractors  may 
charge  a  premium  to  accept  additional  risk  and,  in  some  cases,  may  refuse  the 
contract  altogether.  This  may  happen  if  there  are  too  many  factors  beyond  the 
contractor’s  control  and  the  consequences  of  failure  are  too  high. 

Principal-agent  theory  can  be  applied  to  the  buyer-supplier  relationship  that 
exists  in  defense  acquisitions,  as  well  as  to  the  sponsor-developer  relationship  that 
exists  between  Congress  and  DoD.  Agents  are  responsible  for  accomplishing  the 
work,  but  have  more  information  about  the  uncertainties  of  execution  than  the 
principal.  Program  managers  (PM)  can  find  themselves  acting  as  both  a  principal 
and  an  agent.  When  reporting  to  Congress,  OSD,  and  the  Defense  Acquisition 
Board  (DAB),  the  PM  acts  as  the  agent,  providing  information  on  the  status  of  the 
work  (cost,  schedule  and  performance)  through  programmatic  reviews.  When 
dealing  with  the  contractor  or  supplier,  the  PM  plays  the  role  of  the  principal, 
receiving  information  through  technical  reviews.  The  relationships  are  illustrated  in 
Figure  2. 


Principal 

< - 

Congress 

OSD 

DAB 

Program 

Manager 

Defense 

Programmatic  osRacjj 

Review*  LVKeftartv  VncutsirtY 

Technical  Design  Production 
Reviews  Uncertairty  Uncertainty 

Contraetor 

Supplier 

Principal  <  Agent 


Figure  2.  Principal-agent  Relationships  in  DoD  Acquisitions 
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As  can  be  seen  in  Figure  2,  the  PM  must  respond  to  many  principals.  The 
existence  of  multiple  principals  strongly  indicates  that  not  all  the  principals  will  agree 
on  goals.  Goal  conflict  among  principals  makes  the  relationship  between  principals 
and  agents  exceedingly  complex  (Waterman  &  Meier,  1998).  Each  of  the  principals 
(Congress,  DOD,  and  the  military  services)  has  created  an  intricate  web  of  laws, 
directives,  and  organizations  to  control  the  agents'  actions.  Acquisition  reform  efforts 
have  focused  on  altering  these  rules  and  organizational  roles,  including  increasing 
the  authority  of  certain  players  in  the  acquisition  process  to  alter  the  incentives.  In 
particular,  reformers  designed  initiatives  to  increase  the  information  flow  to  the 
principals  and  to  improve  the  quality  of  the  information  (Biery,  1992). 

TCE  assumes  that  economic  actors — say  government  “principals”  and 
defense  industry  “agents”  in  an  outsourcing  relationship — are  motivated  to  look 
ahead,  recognize  potential  hazards,  and  factor  these  into  contracts  or  organizational 
design.  The  challenge  is  to  design  contracts,  incentive  schemes,  monitoring  and 
enforcement  mechanisms,  and  to  adopt  other  governance  arrangements  (property 
rights,  reputation,  bonding,  warranties,  etc.)  that  allow  for  credible  commitments  ex- 
ante  and  that  promote  mutual  compliance  ex-post  (Williamson,  1983).  Prendergast 
(1999)  provides  a  valuable  overview  of  principal-agent  models  that  highlights  the 
costs  and  consequences  of  various  incentive  mechanisms  designed  to  address 
internal  coordination  and  motivation  issues. 
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III.  The  Program  Manager’s  Perspective 


A.  Program  Oversight 

As  noted  earlier,  the  Department  of  Defense,  for  the  most  part,  commissions 
external  suppliers  to  conduct  projects  for  its  internal  use,  including  the  development 
and  production  of  major  weapon  systems.  Naturally,  there  are  transaction  costs 
associated  with  these  relationships  such  as  search,  information,  decision, 
contracting,  measurement,  monitoring,  and  enforcement  costs. 

DoD  relies  on  a  cadre  of  military  and  civilian  officials — known  as  program 
managers — to  lead  the  development  and  delivery  of  hundreds  of  weapon  systems 
and  subsystems.  The  systems  that  program  managers  are  responsible  for  range 
from  highly  sophisticated  air-,  land-,  sea-,  and  space-based  systems  to  smaller,  less 
complex  communications  or  support  equipment  that  interconnects  or  supports  larger 
systems.  Program  managers  are  responsible  for  assuring  that  these  systems  are 
reliable,  affordable,  supportable,  and  effective.  They  carry  out  multiple  roles  and 
responsibilities  and  are  expected  to  have  a  working  knowledge  in  such  diverse  areas 
as  contracting,  budgeting,  systems  engineering,  and  testing  (GAO,  2005). 

In  addition  to  reporting  to  Congress,  OSD  and  the  DAB  and  monitoring  the 
contractor  or  supplier  (as  shown  in  Figure  2),  program  managers  must  also  work 
with  other  organizations  that  can  influence  the  success  of  their  program.  These 
organizations  include  the  Cost  Analysis  Improvement  Group  (CAIG)  at  OSD  (which 
provides  independent  cost  estimates  to  overcome  moral  hazard  issues),  the 
developmental  and  operational  test  centers,  the  budgeting  and  comptroller 
organizations  that  work  with  funding,  the  ultimate  user  of  the  weapon  system  (the 
warfighter  that  defines  the  desired  capabilities),  as  well  as  other  organizations, 
including  other  services  interested  in  joint  interoperability  issues. 
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As  shown  in  Figure  1  above,  program  managers  report  to  a  Program 
Executive  Officer  (PEO) — a  civilian  at  the  senior  executive  level  or  military  officer  at 
the  general  officer  rank — who  typically  manages  a  portfolio  of  related  weapon 
systems.  The  PEO  reports  to  the  Service  Acquisition  Executive  (SAE),  a  civilian 
(often  a  political  appointee)  who  reports  to  the  service  Secretary.  The  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD/AT&L)  is  the 
Defense  Acquisition  Executive  (DAE)  with  full  responsibility  for  supervising  the 
performance  of  the  DoD  Acquisition  System.  As  the  chair  of  the  Defense  Acquisition 
Board  (DAB),  USD(AT&L)  is  the  final  decision  authority  for  major  defense  acquisition 
programs.  DoD  classifies  acquisition  programs  into  categories  based  upon  a 
number  of  factors  such  as  their  size,  cost,  complexity  and  importance.  Table  1 
shows  the  major  defense  acquisition  categories  along  with  the  corresponding 
decision  authority. 

Table  1.  Major  Defense  Acquisition  Categories  (FY2000  dollars) 

(DoD,  2008,  December  8). 


Acquisition  category 

Definition 

Decision  Authority 

Category  I 

Research,  development,  test,  and 
evaluation  >  $365M 

Procurement  >  $2.1 9B 

ID:  USD(AT&L) 

1C:  Head  of  DoD  Component  or,  if 
delegated,  the  Component 
Acquisition  Executive  (CAE) 

Category  II 

Research,  development,  test,  and 
evaluation  >  $140M 

Procurement  >  $660M 

Component  Acquisition  Executive 
or  individual  designated  by  CAE 

Category  III 

No  fiscal  criteria 

Designated  by  CAE 

In  2005,  GAO  conducted  a  survey  of  acquisition  category  I  and  II  program 
managers  to  gather  information  about  their  perceptions  of  factors  that  assist  or  block 
their  success  and  to  help  define  other  issues  in  the  DoD  acquisition  process  that 
affect  program  manager  effectiveness.  According  to  the  GAO,  many  program 
managers  expressed  frustration  with  the  time  required  of  them  to  answer  continual 
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demands  for  information  from  oversight  officials — many  of  which  did  not  seem  to  add 
value.  Some  program  managers,  in  fact,  estimated  that  they  spent  more  then  50%  of 
their  time  producing  and  tailoring  and  explaining  status  information  to  others.  More 
broadly,  many  program  managers  and  program  executive  officials  said  that  they  did 
not  believe  that  DoD’s  acquisition  process  really  supported  or  enabled  them. 

Instead,  they  viewed  the  process  as  cumbersome  and  the  information  produced  as 
non-strategic  (GAO,  2005). 

In  addition,  GAO  found  that  program  managers  believe  that  they  are  not 
sufficiently  supported  once  programs  begin.  In  particular,  they  believe  that  program 
decisions  are  based  on  funding  needs  of  other  programs  rather  than  demonstrable 
knowledge;  they  lack  tools  needed  to  enable  them  to  provide  leadership  consistent 
with  cost,  schedule  and  performance  information;  they  are  not  trusted;  they  are  not 
encouraged  to  share  bad  news;  and  they  must  continually  advocate  for  their 
programs  in  order  to  sustain  DoD  commitment  (GAO,  2005). 

B.  Program  Reviews 

The  DoD  acquisition  process  is  based  on  a  series  of  milestone  decisions  that 
are  supported  by  different  levels  of  program  reviews  intended  to  give  the  program 
manager  and  DoD  leadership  the  information  they  need  to  make  decisions.  Table  2 
illustrates  the  review  process  over  the  life  of  an  acquisition  program. 
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Table  2.  Example  of  the  Timing  and  Levels  of  Reviews 
over  the  Life  of  a  Defense  Acquisition  Program 


Note:  CD  =  Concept  Development,  CR  =  Concept  Refinement,  A,  B  &  C  =  Milestone  A,  B  &  C, 
LRIP  =  Low  Rate  Initial  Production,  FRP  =  Full  Rate  Production 
(see  appendix  for  a  review  of  applicable  acronyms) 

In  a  study  examining  the  various  iterations  of  the  DoD  5000  Series 
regulations  governing  acquisition  programs,  Dillard  (2005)  noted  that  both  the 
number  and  level  of  reviews  conducted  over  the  years  have  increased  substantially, 
particularly  when  taking  into  account  the  array  of  pre-briefs  and  informational 
meetings  held  in  support  of  the  formal  reviews.  He  observed  that  program  reviews 
of  any  kind  at  the  OSD  level  have  a  significant  impact  on  program  management 
offices.  Much  documentation  must  be  prepared  and  many  preparatory  meetings  are 
conducted  before  the  ultimate  review.  And  while  efforts  to  prepare  for  non-milestone 
reviews  are  generally  considered  to  be  lesser  in  scope,  a  considerable  amount  of 
effort  managing  the  decision  process  is  still  expended  by  the  program  manager. 
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These  findings  were  confirmed  in  a  recent  study  commissioned  by  the  Air 
Force  (NRC,  2009).  A  committee  of  the  National  Academy  of  Sciences — which 
included  one  of  the  co-authors  of  this  report  (John  Dillard) — found  that  the  number  of 
program  reviews  is  growing  and  that  the  reviews  “most  certainly”  add  to  program 
costs.  Of  most  concern  to  the  committee  was  that  “the  proliferation  of  reviews  does 
not  appear  to  have  had  a  positive  effect  on  program  cost  and  schedule  outcomes.” 
The  committee  also  recognized  that  there  is  a  significant  amount  of  preparation  and 
coordination  required  for  reviews — both  vertically  in  their  conduct  at  multiple  levels  of 
responsibility,  and  horizontally  across  adjacent  staff  offices. 

The  program  manager  plays  a  central  role  in  the  acquisition  process, 
participating  in  all  reviews  (and  pre-briefs)  for  multiple  principals.  All  of  the  reviews, 
both  formal  and  informal,  must  be  supported  by  the  program  management  office  and 
require  substantial  resources.  Not  only  government  resources  are  expended  in 
preparing  and  presenting  reviews,  but  often  contractor  personnel  are  involved  as 
well.  Although  each  individual  review  is  intended  to  serve  a  specific  purpose,  the 
overall  magnitude  of  the  review  efforts  not  only  significantly  increase  the  workload  of 
the  program  office  in  terms  of  direct  support,  but  also  divert  attention  from  day-to- 
day  management  of  the  program  (NRC,  2009). 

There  are  two  basic  categories  of  program  reviews:  technical  and 
programmatic.  Technical  reviews  are  focused  on  the  performance  of  the  system 
being  acquired,  while  programmatic  reviews  are  more  oversight  in  nature. 

The  Defense  Acquisition  University  (DAU)  defines  a  technical  review  as  an 
event  at  which  the  progress  of  the  technical  effort  is  assessed  relative  to  its 
governing  plans  and  technical  requirements.  Technical  reviews  are  key  decision 
events  used  to  measure  technical  progress  and  maturity  in  system  development. 

DoD  Instruction  5000.02,  Enclosure  12,  paragraph  4  (DoD,  2008,  Decembers), 
directs  that  technical  reviews  of  program  progress  be  event  driven  and  conducted 
when  the  system  under  development  meets  the  review  entrance  criteria  as 
documented  in  the  Systems  Engineering  Plan. 
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The  Interim  Defense  Acquisition  Guidebook  (DAU,  2009),  chapter  4,  section 
4. 3. 3.4,  lists  the  following  technical  reviews  as  part  of  the  systems  engineering 
process: 

■  Technology  development  phase; 

-  System  Requirements  Review 

-  System  Functional  Review 

-  System  Preliminary  Design  Review 

■  Engineering  and  manufacturing  development  phase: 

-  Integrated  Baseline  Review 

-  Critical  Design  Review 

-  Test  Readiness  Review 

-  Flight  Readiness  Review 

-  System  Verification  Review 

-  Functional  Configuration  Audit 

-  Production  Readiness  Review 

-  Technology  Readiness  Assessment 

Technical  reviews  serve  as  forums  for  problem  discovery  and  assessment  of 
technical  progress  toward  system  performance  goals.  Sharing  of  information 
horizontally  is  key  in  these  integrative  reviews.  They  are  typically  chaired  by  the 
program  manager  and  conducted  locally — often  in  the  prime  or  system  contractor 
location — and  are  scheduled  and  conducted  at  the  program  manager’s  discretion 
when  appropriate  (event  or  progress-based  vs.  calendar-based).  Technical  reviews 
are  also  used  as  oversight  tools  by  the  program  manager  (as  principal)  to  monitor 
the  technical  progress  of  the  system  prime  contractor  and  subcontractors  (agents). 

DAU  defines  “programmatic”  as  pertaining  to  the  cost,  schedule,  and 
performance  characteristics  of  an  acquisition  program  (2005).  This  definition  would 
distinguish  programmatic  reviews  from  technical  reviews  in  that  technical  reviews 
are  primarily  concerned  with  performance  issues  of  the  system,  while  programmatic 
reviews  are  concerned  with  the  performance  of  the  acquisition  process. 
Programmatic  reviews  can  be  thought  of  as  those  reviews  dealing  with  non-technical 
issues  such  as  cost,  schedule,  budget,  quantities,  contracts  and  program 
management,  as  well  as  performance.  Programmatic  reviews  consider  performance 
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issues,  but  at  a  higher  level  and  in  the  context  of  the  overall  program  management 
process. 

Although  programmatic  reviews  are  often  scheduled  and  conducted 
according  to  prearranged  milestone  dates  or  per  an  “acquisition  baseline” 
established  by  program  manager  and  multi-level  staffs  well  in  advance,  they  can  be 
held  “on  call”  or  ad  hoc  as  program  issues  arise.  They  are  formal  in  nature, 
convened  and  chaired  by  higher  echelons  (program  executive  officer,  service-level 
acquisition  executive.  Office  of  the  Secretary  of  Defense  staff,  and  Milestone 
Decision  Authority/Defense  acquisition  executive),  and  information  within  them  flows 
vertically.  Programmatic  reviews  are  designed  to  inform  key  staff  members  in  the 
Pentagon  and  Congress  (the  principals)  about  the  business  and  technical  progress 
of  programs.  Decisions  based  on  programmatic  reviews  can  determine  the  fate  of  a 
program — allowing  it  to  proceed  on  course,  sometimes  directing  a  change  in  course, 
or  in  some  cases,  catalyzing  the  termination  of  the  program.  These  types  of  reviews 
have  increased  significantly  in  both  frequency  and  in  their  level  of  management 
oversight. 

In  2005,  GAO  noted  that  DOD  program  managers  operate  under  many  layers 
of  oversight — both  internally  and  externally.  While  much  of  the  oversight  is 
necessary  for  carrying  out  stewardship  responsibilities  for  public  money,  GAO’s 
researchers  pointed  out  that  studies  conducted  by  a  variety  of  commissions 
assessing  acquisition  problems  through  the  years  have  consistently  found  that  there 
are  opportunities  to  reduce  oversight  layers  and  streamline  oversight  processes. 
Program  managers  (as  agents)  understand  the  need  for  oversight,  but  responding  to 
oversight  demands  can  take  too  much  of  their  time.  The  next  section  discusses  the 
findings  of  a  recent  study  that  tried  to  address  the  same  issues. 
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C.  National  Research  Council  Survey 


In  2008,  the  Deputy  Assistant  Secretary  of  the  Air  Force  for  Science, 
Technology,  and  Engineering  asked  the  National  Research  Council  (NRC)  to  (1) 
review  and  assess  the  increasing  number  of  prescribed  program  reviews  and 
assessments  that  US  Air  Force  space  and  non-space  acquisition  programs  in  all 
Department  of  Defense  (DoD)  acquisition  categories  are  required  to  undergo,  and 
(2)  to  recommend  ways  to  improve  the  effectiveness  and  efficiency  of  those  program 
reviews  in  terms  of  their  goals,  objectives,  content,  and  requirements.  The 
Committee  on  Optimizing  US  Air  Force  and  DoD  Review  of  Air  Force  Acquisition 
Programs  was  formed  in  May  2008  to  conduct  this  review  (NRC,  2009). 

The  committee  was  asked  to  address  a  key  question:  Can  changes  in  the 
number,  content,  sequence,  or  conduct  of  program  reviews  help  program  managers 
more  successfully  execute  their  programs?  Specifically,  the  committee  was  tasked 
by  the  Air  Force  to:  review  the  program  management  and  the  technical  reviews  and 
assessments  that  US  Air  Force  space  and  non-space  system  acquisition  programs 
are  required  to  undergo;  assess  each  review  in  terms  of  resources  required  and  its 
role  and  contribution;  identify  cases  in  which  different  reviews  have  common  or 
overlapping  goals,  content,  or  requirements;  identify  and  evaluate  options  for 
streamlining,  tailoring,  integrating,  or  consolidating  reviews  of  programs  to  increase 
cost-effectiveness  and  to  lessen  the  workforce  impact  of  the  reviews  as  a  whole;  and 
recommend  changes  that  the  Air  Force  and  the  Department  of  Defense  should 
make. 


Because  data  and  metrics  on  program  reviews  are  not  collected  by  DoD,  the 
committee  was  unable  to  determine  the  overlap  or  duplication  of  different  reviews. 
The  committee  was  also  unable  to  determine  specific  resources  required  to 
accomplish  each  review,  although  answers  to  the  survey  and  information  gathered 
from  the  interviews  suggested  that  10-30%  of  a  PM’s  time  is  spent  supporting 
reviews. 
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The  committee  gathered  information  from  presentations,  interviews,  and 
previous  studies.  It  also  developed  a  detailed  qualitative  survey  that  went  out  to 
over  80  USAF  program  managers  to  gauge  their  participation  in  and  overall 
assessment  of  both  programmatic  and  technical  reviews,  focusing  upon  the  value  as 
well  as  the  costs  of  reviews. 

In  surveys  and  interviews  conducted  by  the  GAO  in  2005,  program  managers 
and  program  executive  officers  frequently  commented  that  they  spend  too  much 
time  preparing  data  for  oversight  purposes  that  are  not  strategic  or  very  useful  to 
them.  GAO  noted  that  more  than  90%  of  survey  respondents  said  they  spent  either 
a  moderate,  great,  or  very  great  extent  of  their  time  representing  their  program  to 
outsiders  and  developing  and  generating  information  about  program  progress. 

After  reviewing  studies  conducted  over  the  past  decade,  the  committee 
reported  that  it  could  find  no  evidence  of  earlier  work  that  focused  on  the  impact  of 
the  overall  formal  and  informal  review  process  on  the  acquisition  system  in  terms  of 
resources  spent  by  the  program  office  or  the  effect  of  diverting  a  program  manager’s 
attention  from  the  day-today  management  of  his  or  her  programs.  Accordingly,  the 
committee  decided  to  examine  the  costs  of  reviews  in  terms  of  the  amount  of  time 
the  program  manager  spends  supporting  reviews. 

The  committee  surveyed  Air  Force  PMs  and  PEOs  to  collect  quantitative  and 
qualitative  information  on  the  impact  of  external  reviews  on  program  execution — 
including  the  time  and  effort  spent  preparing  for,  participating  in,  and  following  up  on 
actions  resulting  from  tasks  from  higher-level  AF  and  OSD  reviews — that  would  not 
otherwise  have  had  to  be  spent  for  the  purpose  of  good  program  management.  The 
survey  also  asked  about  PM  and  PEO  assessments  of  the  value  of  the  reviews. 
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The  survey  was  divided  into  four  sections: 


Section  1 :  Demographic  Data  (information  on  program  manager  and 
program) 

Section  2:  Program  Activity  Overview  (information  on  pertinent  external 
reviews/reporting  accomplished  by  the  program) 

Section  3:  Questions  on  Specific  Reviews  (information  on  time/effort  spent 
on  specific  reviews/reporting  accomplished  by  each  program 
manager  taking  the  survey) 

Section  4:  Optional  Section  (to  comment  on  streamlining,  tailoring, 
integrating  and  consolidating  opportunities) 

The  committee  concluded  that  there  may  not  be  sufficient  data  to  permit  a 
quantitative  response  to  the  key  question  raised  in  the  summary — namely,  can 
changes  in  the  number,  content,  sequence,  or  conduct  of  program  reviews  help  the 
program  manager  more  successfully  execute  the  program?  Instead,  the  committee 
made  five  recommendations  which  it  believes  will  provide  greater  control  of  the 
review  process. 

D.  Implications  of  Theory 

Transaction  cost  economics  and  the  principal-agent  model  suggest  several 
interesting  questions  that  might  be  addressed  by  analyzing  the  data  gathered  in  the 
NRC  PM  survey: 

1 .  Can  we  use  the  PM  survey  to  quantify  oversight  and  monitoring  costs? 

2.  Is  there  a  difference  in  the  perceived  cost/benefit  of  oversight  and 
monitoring  activities  when  the  program  manager  acts  as  the: 

a.  Principal  (technical  reviews)? 

b.  Agent  (programmatic  reviews)? 
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This  study  attempts  to  answer  these  questions.  In  our  analysis,  we  assume 
that  subjective  perceptions  of  relative  cost/benefit  of  reviews  provide  useful  insights 
into  transaction  costs.  We  start  with  the  assumption  that  program  managers  are 
more  likely  to: 

■  View  technical  reviews  as  less  costly/more  beneficial 

■  View  programmatic  reviews  as  more  costly/less  beneficial 

The  results  of  our  analysis  are  presented  in  the  next  section. 
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IV.  Data  Analysis 


A.  Survey  Data 

The  data  used  for  this  analysis  is  based  on  responses  to  the  AF  NRC 
Program  Manager  Survey  (NRC,  2009).  The  only  data  available  from  the  survey 
report  is  in  summary  form  (i.e.,  the  total  response  count  for  each  question  is 
available  but  not  the  individual  responses).  We  limited  our  analysis  to  Section  2  of 
the  survey — Program  Activity  Overview — which  provided  information  on  pertinent 
external  reviews/reporting  accomplished  by  programs.  This  was  the  only  section  of 
the  survey  that  lent  itself  to  meaningful  analysis  given  the  limitations  of  summarized 
data. 

Each  of  the  questions  selected  for  our  analysis  lists  the  nine  reviews  in  Table 
1  and  asks  the  respondent  to  answer  in  terms  of  those  reviews.  Survey  participants 
were  also  allowed  to  write-in  other  reviews,  but  to  make  the  data  more  comparable, 
we  did  not  use  the  write-in  answers  in  our  analysis.  We  divided  the  survey  data  into 
two  categories:  technical  reviews  and  programmatic  reviews  (as  shown  in  Table  3). 
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Table  3.  Technical  and  Programmatic  Reviews 


Technical 

Programmatic 

Technology  Readiness 

Assessment  (TRA) 

Defense  Acquisition  Board  (DAB)  Milestone 
Review 

Technology  Maturity  Assessment 
(TMA) 

Defense  Space  Acquisition  Board  (DSAB) 
Milestone  Review 

Manufacturing  Readiness  Review 
(MRR) 

Defense  Acquisition  Board  (DAB)  Status 
Review 

System  Engineering  Assessment 
Model  (SEAM) 

Defense  Space  Acquisition  Board  (DSAB) 
Status  Review 

Qverarching  Integrated  Product  Team  (QIPT) 
Review 

Independent  Program  Assessment  (IPA)  * 

Program  Support  Review  (PSR) 

Logistics  Health  Assessment  (LHA) 

Air  Force  Review  Board  (AFRB) 

*  The  AF  study  categorized  I  PA  as  technical,  but  our  subject-matter  expert  felt  that 
the  I  PA  had  enough  business  aspects  to  be  more  accurately  characterized  as 
Programmatic. 


The  questions  selected  for  our  analysis  are  the  following: 

For  each  of  these  major  program  reviews/assessments  that  your 
program  experienced,  indicate  your  assessment  of  their  impact  on 
program  performance  (i.e.,  cost/schedule/  technical  performance 
accomplishment)? 

Which  single  review  had  the  greatest  positive  impact  on  program 
performance? 

Higher-level  HQ  AF/OSD  reviews/assessments  provide  senior  leaders 
information  that  is  necessary  for  their  understanding  of  program 
performance,  to  fulfill  their  oversight  role.  Please  rate  each  of  the 
reviews  that  your  program  experienced  in  terms  of  how  effective  you 
believe  the  structure/format  of  the  review  was  at  providing  useful  data 
to  the  senior  AF  and  OSD  leadership. 

From  the  list  below,  identify  the  three  higher-level  HQ  AF/OSD 
reviews/reporting  activities  that  you  believe  have  the  LEAST  beneficial 
impact  on  program  performance.  Respondents  indicated  the  least, 
second-least  and  third-least  beneficial. 
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2.2 

2.2a 

2.4 

2.5 


2.7  From  what  you  know  from  any  source,  identify  the  program  reviews 
that  have  the  highest  potential  to  be  combined  into  a  single  useful 
review.  Respondents  were  asked  to  select  from  the  list  of  reviews  in 
Table  1  and  use  the  write-in  section  to  show  the  pairings/groupings. 
Only  a  few  of  the  respondents  used  the  write-in  section,  so  it  was  not 
part  of  our  analysis. 

2.12  For  the  following  major  reviews,  please  indicate  your  opinion  about 
whether  the  documentation  required  by  higher  authorities  to  support 
each  of  the  following  reviews  is  Insufficient  (In),  About  Right  (AR), 
Excessive  but  Decreasing(E-D),  Excessive  and  Steady  (E-S), 
Excessive  and  Increasing  (E-l).  For  purposes  of  our  analysis,  E-D,  E- 
S  and  E-l  were  grouped  into  one  category:  Excessive. 

B.  Hypotheses 

The  following  hypotheses  are  tested  (the  survey  question  corresponding  to 
each  hypothesis  is  shown  in  parentheses): 

1 .  The  perceived  value  (impact)  of  technical  reviews  is  higher  than  the 
value  of  program  reviews.  (2.2  and  2.2a) 

2.  Technical  reviews  are  more  likely  to  be  rated  as  helpful  (provide  useful 
data)  than  program  reviews.  (2.4) 

3.  Technical  reviews  are  more  likely  to  be  perceived  as  beneficial  than 
program  reviews.  (2.5) 

4.  Technical  reviews  are  more  likely  to  be  perceived  as  well  structured 
(less  likely  to  be  combined  with  other  reviews)  than  program  reviews. 
(2.7) 

5.  The  perceived  cost  (level  of  documentation  required)  of  program 
reviews  is  significantly  higher  than  the  cost  of  technical  reviews.  (2.12) 

C.  Statistical  Analysis 

The  proportions  (relative  frequency)  of  responses  in  the  two  categories 
(technical  and  programmatic)  are  examined  in  two  ways: 

A  Chi-squared  test  is  performed  on  the  contingency  table  for  each 
question  (where  applicable)  to  determine  if  the  counts  in  the  rows 
(answers)  and  columns  (review  type)  can  be  considered  independent. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY  -  25  - 

NAVAL  POSTGRADUATE  SCHOOL 


b)  A  z-test  of  the  difference  between  the  proportions  in  each  category  is 
used  to  determine  if  there  is  statistically  significant  difference  in  the 
proportions  at  the  .05  level.  A  one-tailed  test  is  used  to  determine  if 
the  proportion  of  technical  responses  is  significantly  higher  than  the 
proportion  of  programmatic  responses.  The  one-tailed  test  of  the 
reverse  (proportion  of  programmatic  responses  is  greater  than 
technical  responses)  is  also  shown. 

c)  A  z-test  of  the  difference  between  the  proportion  of  technical 
responses  vs.  the  expected  frequency  (based  on  the  number  of 
technical  reviews  in  the  survey,  4  out  of  13  or  p  =  .3077)  is  used  to 
determine  if  the  level  of  responses  is  statistically  significant  at  the  .05 
level. 

The  results  of  the  tests  are  summarized  in  Table  4. 

Table  4.  Summary  of  Statistical  Test  Results 

(Aggregated  responses  are  shown  in  italics) 


Question 

Technical 

Programmatic 

Chi-2 

p-value 

one-tailed 

T>P 

p-value 

one-tailed 

P>T 

p-value 

one-tailed 

T>4/13 

p-value 

2.2 

0.0939 

Positive  Impact 

20 

46 

0.8402 

0.1598 

0.5328 

No  Impact 

27 

34 

0.0205 

0.9795 

0.0112 

Negative 

Impact 

5 

18 

0.9216 

0.0784 

0.8260 

Positive  or  No 
Impact 

47 

80 

0.0784 

0.9216 

0.0639 

2.2a 

10 

27 

0.3955 

2.4 

0.0907 

Lots  of  Useful 
Data 

14 

25 

0.3720 

0.6280 

0.2439 

Some  Useful 
Data 

19 

58 

0.9892 

0.0108 

0.8767 

Little  Useful 
Data 

14 

18 

0.0912 

0.9088 

0.0558 
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Question 

Technical 

Programmatic 

Chi-2 

p-value 

one-tailed 

T>P 

p-value 

one-tailed 

P>T 

p-value 

one-tailed 

T>4/13 

p-value 

No  Useful  Data 

9 

9 

0.0611 

0.9389 

0.0386 

Lots  or  Some 
Useful  Data 

33 

83 

0.9859 

0.0141 

0.7060 

Little  or  No 
Useful  Data 

23 

27 

0.0141 

0.9859 

0.0098 

2.4A 

0.0572 

Lots  of  Useful 
Data 

8 

31 

0.9754 

0.0246 

0.9174 

Some  Useful 
Data 

19 

44 

0.7571 

0.2429 

0.5419 

Little  Useful 
Data 

12 

14 

0.0640 

0.9360 

0.0446 

No  Useful  Data 

12 

13 

0.0445 

0.9555 

0.0310 

Lots  or  Some 
Useful  Data 

27 

75 

0.9946 

0.0054 

0.8266 

Little  or  No 
Useful  Data 

24 

27 

0.0054 

0.9946 

0.0500 

2.5 

0.4382 

Least 

Beneficial 

17 

23 

0.5014 

0.4986 

0.0540 

Second-least 

Beneficial 

18 

18 

0.8657 

0.1343 

0.0062 

Third-least 

Beneficial 

13 

24 

0.1353 

0.8647 

0.2825 

Least  or 
second-least 

35 

41 

0.1353 

0.8647 

<  0.0001 

2.7 

66 

74 

<  0.0001 

2.12 

0.1997 

Insufficient 

6 

5 

0.0650 

0.9350 

0.0438 

About  Right 

42 

76 

0.2542 

0.7458 

0.1281 

Excessive 

40 

94 

0.8969 

0.1031 

0.5912 

Insufficient  or 
About  Right 

48 

81 

0.1031 

0.8969 

0.0565 
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D.  Results  of  Statistical  Analysis 


The  significant  results  of  the  analysis  are  discussed  for  each  of  the  questions: 

2.2  Technical  reviews  are  significantly  more  likely  to  be  seen  as  having  no 
impact  on  program  performance  and  are  somewhat  less  likely  to  be 
seen  as  having  a  negative  impact  on  program  performance. 

2.4  There  were  two  set  of  responses  for  question  2.4  (both  are  shown  in 
Table  2).  Both  sets  indicate  that  programmatic  reviews  are 
significantly  more  likely  to  be  seen  as  providing  some  or  lots  of  useful 
data,  while  technical  reviews  are  significantly  more  likely  to  be  seen  as 
providing  little  or  no  useful  data. 

2.5  Of  the  reviews  identified  as  being  least  or  second  least  beneficial,  the 
proportion  of  technical  reviews  is  significantly  higher  than  expected 
based  on  the  number  of  technical  reviews  in  the  survey — indicating 
that  technical  reviews  are  more  likely  to  be  seen  as  least  or  second- 
least  beneficial. 

2.7  Of  the  reviews  identified  as  having  the  highest  potential  for  being 
combined  into  one  review,  the  proportion  of  technical  reviews  is 
significantly  higher  than  the  proportion  of  technical  reviews  in  the 
survey — indicating  that  technical  reviews  are  more  likely  to  be 
identified  as  candidates  for  consolidation. 

2.12  Of  the  technical  reviews  identified  as  requiring  insufficient 

documentation,  the  proportion  of  technical  reviews  is  significantly 
higher  than  the  proportion  of  reviews  in  the  survey — indicating  that 
technical  reviews  are  more  likely  to  be  seen  as  not  having  enough 
documentation. 

E.  Interpretation  of  Results 

Based  on  our  initial  analysis,  we  can  draw  the  following  conclusions  from  the 


Program  managers  do  not  see  significantly  more  value  in  technical 
reviews  than  they  see  in  programmatic  reviews.  However,  they  do 
seem  to  feel  that  technical  reviews  are  somewhat  less  harmful  (have 
less  of  a  negative  impact  on  program  performance)  than  programmatic 
reviews.  When  acting  as  principals  in  technical  reviews,  they  probably 
see  the  review  as  necessary  for  making  sure  the  program  stays  on 
course;  thus,  it  should  have  a  positive  impact  on  program  performance. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY  -  28  - 

NAVAL  POSTGRADUATE  SCHOOL 


test  results: 
1. 


On  the  other  hand,  the  programmatic  reviews  may  be  more  likely  to 
reveal  negative  information  such  as  cost  overruns  or  schedule  delays 
that  have  a  negative  impact  on  program  performance.  In  addition, 
when  they  are  acting  as  agents  providing  information  to  senior 
leadership,  the  program  managers  may  feel  that  programmatic  reviews 
are  more  likely  to  expose  the  program  to  higher-level  criticism  or 
interference. 

2.  Program  managers  were  asked  to  rate  the  usefulness  of  information 
provided  by  reviews  to  senior  leaders.  Thus,  they  answered  question 
2.4  from  the  perspective  of  the  senior  leadership.  From  this 
perspective,  it  makes  sense  that  senior  leaders  would  find 
programmatic  reviews  more  useful  and  technical  reviews  less  useful. 
Senior  leadership  is  more  interested  in  the  overall  program 
performance,  including  cost  and  schedule  as  well  as  technical  issues. 
At  the  higher  levels  of  OSD  and  AF,  the  technical  issues  are  left  to  the 
program  manager  to  fix;  they  only  become  important  when  they 
significantly  impact  the  overall  performance  of  the  program. 

3.  In  terms  of  reporting  to  higher-level  authorities,  program  managers  see 
technical  reviews  as  providing  less  benefit  to  their  programs.  This 
makes  sense  given  the  previous  finding  that  the  information  provided 
by  technical  reviews  is  less  useful  to  senior  leaders  than  the 
information  in  programmatic  reviews.  Less-useful  data  leads  to  lower 
impact  and  less  benefit  to  the  program. 

4.  Given  that  program  managers  see  the  information  in  technical  reviews 
as  being  less  useful  to  senior  leaders,  it  makes  sense  that  they  would 
identify  more  technical  reviews  as  those  needing  to  be  consolidated — 
perhaps  to  increase  the  usefulness  of  the  information  provided,  or 
perhaps  to  simply  reduce  the  amount  of  information  reported  and  make 
the  reviews  more  efficient. 

5.  Program  managers  believe  higher-level  authorities  do  not  require 
sufficient  documentation  for  technical  reviews.  This  may  be  related  to 
the  usefulness  of  the  information.  Perhaps  more  documentation  is 
required  to  properly  explain  and  illustrate  the  technical  issues  so  that 
higher-level  authorities  can  fully  appreciate  them.  Or  it  may  be  that 
program  managers  are  much  more  involved  in  managing  the  technical 
issues  and,  therefore,  are  more  aware  of  ways  to  document  and 
support  technical  reviews  vs.  programmatic  issues. 
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V.  Conclusion  and  Further  Research 


This  study  continued  ongoing  efforts  by  the  authors  to  understand  and 
measure  transactions  costs  within  DoD  acquisitions.  The  principle-agent  model  was 
presented  to  help  understand  the  program  manager’s  role  in  defense  acquisitions. 
We  analyzed  data  from  an  NRC  survey  of  USAF  program  managers  to  gain  insights 
into  oversight  and  monitoring  costs  both  when  the  program  manager  acts  as  a 
principal  and  as  an  agent. 

One  of  the  interesting  observations  from  our  analysis  of  the  NRC  survey  was 
that  program  managers  found  some  real  value  in  some  of  their  programmatic 
reviews,  despite  the  fact  that  such  reviews  are  usually  viewed  as  excessive  and 
burdensome  levels  of  oversight.  While  surely  much  will  be  relative  to  individual 
project  circumstances,  program  managers  may  find  themselves  genuinely  pleased 
and  relieved  when  such  reviews  are  over  and  the  needed  decisions  have  been 
obtained.  Insofar  as  they  may  be  spending  more  time  on  the  decision  bureaucracy 
than  overseeing  the  technical  and  business  efforts  of  their  program,  this 
management  of  government  requirements  and  resources  is,  nonetheless,  essential 
to  program  success.  Thus,  when  bureaucratic  reviews  for  decision-making  become 
as  critical  as  other  items  of  scope  in  the  project,  they  can  easily  be  perceived  as 
valuable  when  completed.  In  essence,  programmatic  reviews  (being  off-core 
activities  to  the  more  central  scope  of  the  project)  evolve  into  actual  scope — with  a 
budget  (or  at  least  a  cost)  and  a  schedule  to  accomplish  each  review,  and  with 
measures  of  performance  (or  effectiveness)  dependent  upon  their  outcome. 

The  lesser  value  given  to  technical  reviews  was  counterintuitive.  Our 
expectation  was  that  since  program  managers  use  the  information  in  technical 
reviews  on  a  day-to-day  basis,  they  would  find  such  reviews  more  useful  than 
programmatic  reviews.  A  possible  explanation  is  that  they  are  typically  deliverable 
products  under  the  program’s  system  development  contract,  and  the  measure  of 
quality  in  performance  might  vary  between  programs  depending  upon  how  much 
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emphasis  the  government-industry  management  team  applies  to  them.  While  the 
program  manager  typically  chairs  each  review,  the  presentations  are  made  by  the 
system  prime  contractor  and  his  subcontractors.  If,  in  fact,  technical  reviews  serve  to 
usefully  inform  the  development  team  at  the  various  points  at  which  they  are 
conducted,  they  will  have  served  their  intended  purpose.  But  if,  in  fact,  the  reviews 
are  conducted  as  mere  formalities  to  “check  the  block”  on  their  fulfillment,  they  will 
be  regarded  as  such — particularly  if  the  program  manager  is  well  aware  of  the 
issues  presented. 

Unfortunately,  we  were  not  given  access  to  the  raw  data  gathered  through  the 
NRC  survey,  only  summarized  responses.  Without  the  raw  data,  it  was  impossible 
to  analyze  important  information  about  specific  reviews  gathered  in  Section  3  of  the 
survey.  Section  3  asked  the  respondent  to  answer  more  questions  for  a  selected 
review  (based  on  answers  to  Section  2).  This  is  the  most  interesting  set  of  answers, 
but  basically  useless  in  summary  form  because  a  reader  can’t  determine  to  which 
review  the  answers  correspond. 

The  raw  data  would  allow  us  to  do  further  research  into  the  cost/benefit  of 
specific  reviews — such  research  may  provide  insights  into  the  effectiveness  of 
incentives  and  the  principal-agent  relationships  in  DoD  acquisitions. 
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Review  Acronyms 


AFRB 

Air  Force  Review  Board 

ASP 

Acquisition  Strategy  Panel 

ASR 

Alternative  System  Review 

CDR 

Critical  Design  Review 

CSB 

Configuration  Steering  Board 

DRR 

Design  Readiness  Review 

IBR 

Integrated  Baseline  Review 

ITR 

Initial  Technical  Review 

FCA 

Functional  Configuration  Audit 

OIPT 

Overarching  Integrated  Product  Team 

OTRR 

Operational  Test  Readiness  Review 

PDR 

Preliminary  Design  Review 

PSR 

Program  Support  Review 

SFR 

System  Functional  Review 

SR 

System  Review 

SRR 

System  Requirements  Review 

TRA 

Technology  Readiness  Assessment 

TRR 

Test  Readiness  Review 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-33- 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


List  of  References 


Alchian  A.,  &  Demsetz,  H.  (1972).  Production,  information  costs  and  economic 
organization.  American  Economic  Review,  62,  777-795. 

Angelis,  D.,  Dillard,  J.,  Franck,  R.,  &  Melese,  F.  (2007).  Applying  insights  from 
transaction  cost  economics  (TCE)  to  improve  DoD  cost  estimation.  In 
Proceedings  of  the  fourth  annual  acquisition  research  symposium.  Monterey, 
CA;  Naval  Postgraduate  School.  Retrieved  October  1, 2009,  from 
http://acquisitionresearch.net/_files/FY2007/NPS-AM-07-004.pdf 

Angelis,  D.,  Dillard,  J.,  Franck,  R.,  &  Melese,  F.  (2008).  Measuring  transaction  costs 
in  DoD  acquisition  programs  (NPS-AM-08-126).  Monterey,  CA;  Naval 
Postgraduate  School.  Retrieved  October  1, 2009,  from 
http://acquisitionresearch.net/_files/FY2008/NPS-AM-08-126.pdf 

Angelis,  D.,  Dillard,  J.,  Franck,  R.,  Melese,  F.,  Brown,  M.,  &  Flowe,  R.  (2008) 

Application  of  transaction  cost  economics  to  capabilities-based  acquisition: 
Exploring  single  service  vs.  joint  service  programs  and  single  systems  vs. 
system-of-systems.  In  Proceedings  of  the  fifth  annual  acquisition  research 
symposium.  Monterey,  CA:  Naval  Postgraduate  School.  Retrieved  October  1, 
2009,  from  http://acquisitionresearch.net/_files/FY2008/NPS-AM-08-023.pdf 

Biery,  F.  (1992).  The  effectiveness  of  weapon  system  acquisition  reform  efforts. 
Journal  of  Policy  Analysis  and  Management,  11  {4),  637-  664. 

Brown,  M.,  Flowe,  R.,  &  Flamel,  S.  (2007).  The  acquisition  of  joint  programs:  The 
implications  of  interdependence.  CrossTalk — The  Journal  of  Defense 
Software  Engineering,  20(5),  20-24. 

Coase,  R.  (1937).  The  nature  of  the  firm.  Economica,  4,  386-405. 

Defense  Acquisition  University  (DAU).  (2005).  Programmatic.  In  Glossary  of  defense 
acquisition  acronyms  and  terms.  Retrieved  October  1 ,  2009,  from 
http://www.dau.mil/pubs/qlossarv/12th  Glossary  2005.pdf 

Defense  Acquisition  University  (DAU).  (2009).  Interim  Defense  Acquisition 
Guidebook.  Retrieved  October  1 ,  2009,  from  https://acc.dau.mil/daq 

Demski,  J.,  &  Feltham,  G.  (1978).  Economic  incentives  in  budgetary  control 
systems.  Accounting  Review,  53,  336-359. 

Dillard,  J.  (2005,  August-November).  Toward  centralized  control  of  defense 

acquisition  programs.  Defense  Acquisition  Review  Journal,  12(3),  330-344. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-35- 


Franck,  R.,  &  Melese,  F.  (2005).  A  transaction  cost  economics  view  of  DoD 
outsourcing.  In  Proceedings  of  second  annual  acquisition  research 
symposium.  Monterey,  CA:  Naval  Postgraduate  School.  Retrieved  October  1, 
2009,  from  http://acquisitionresearch.net/_files/FY2005/NPS-AM-05-004.pdf 

Franck,  R.,  Melese,  &  F.  Dillard,  J.(2006).  A  transaction  cost  economics  approach  to 
defense  acquisition  management.  In  Proceedings  of  the  third  annual 
acquisition  research  symposium.  Monterey,  CA:  Naval  Postgraduate  School. 
Retrieved  October  1 ,  2009,  from 

http.7/acquisitionresearch.net/_files/FY2006/NPS-AM-06-01 1  .pdf 

Government  Accountability  Office  (GAO).  (2005).  Better  support  of  weapon  systems 
managers  needed  to  improve  outcomes  (GAO-06-1 1 ).  Report  to  the 
Subcommittee  on  Readiness  and  Management  Support,  Committee  on 
Armed  Services,  US  Senate.  Washington,  DC:  Author. 

Jensen,  M.,  &  Meckling,  W.  (1976).  Theory  of  the  firm:  Managerial  behavior,  agency 
costs,  and  ownership  structure.  Journal  of  Financial  Economics,  3,  305-360. 

Melese,  F.,  Franck,  R.,  Angelis,  D.,  &  Dillard,  J.  (2007).  Applying  insights  from 
transaction  cost  economics  to  improve  cost  estimates  for  public  sector 
purchases:  The  case  of  U.S.  military  acquisition.  International  Public 
Management  Journal,  10{4),  357-385. 

National  Research  Council  (NRC).  (2009).  Optimizing  U.S.  Air  Force  and 
Department  of  Defense  review  of  Air  Force  acquisition  programs. 

Washington,  DC:  National  Academies  Press.  Retrieved  October  1, 2009,  from 
http://www.nap.edu/catalog/12673.html 

Pint,  E.,  &  Baldwin,  L.  (1997).  Strategic  sourcing:  Theory  and  evidence  from 

economic  and  business  management  (Mr-865-Af).  Santa  Monica,  CA:  RAND. 

Prendergast,  C.  (1999).  The  provision  of  incentives  in  firms.  Journal  of  Economic 
Literature,  37(1),  7-63. 

Ross,  S.  (1973).  The  economic  theory  of  agency:  The  principal's  problem.  American 
Economic  Review,  63,  134-139. 

Spring,  B.  (2002,  May).  Don’t  let  politics  or  bureaucracy  hobble  missile  defense 

(Executive  Memorandum  817).  The  Fleritage  Foundation.  Retrieved  October 
1 ,  2009,  from 

https://www.policyarchive.org/bitstream/handle/10207/8296/em_817.pdf 

Under  Secretary  of  Defense  (Acquisition,  Technology  &  Logistics)  USD(AT&L). 
(2008,  December  8).  Operation  of  the  defense  acquisition  system 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-36- 


(DoD  Instruction  5000.02).  Retrieved  October  1,  2009,  from 
http://www.dtic.mil/whs/directives/corres/pdf/500002p.pdf 

Wang,  N.  (2003).  Measuring  transaction  costs:  An  incompiete  survey  (Ronald 
Coase  Institute  Working  Paper,  No  2).  Retrieved  October  1,  2009,  from 
http.7/www. coase.org/workingpapers/wp-2.pdf 

Waterman,  R.,  &  Meier,  K.  (1998,  April).  Principal-agent  models:  An  expansion? 
Journai  of  Pubiic  Administration  Research  and  Theory,  8(2),  173-202  . 

Williamson,  O.  (1971,  May).  The  vertical  integration  of  production:  Market  failure 
considerations.  American  Economic  Review,  61,  112-123. 

Williamson,  O.  (1979).  Transaction-cost  economics:  The  governance  of  contractual 
relations.  Journai  of  Law  and  Economics,  22,  233-261. 

Williamson,  O.  (1983).  Organization  form,  residual  claimants  and  corporate  control. 
Journai  of  Law  and  Economics,  36,  351-66. 

Williamson,  O.  (1999).  Public  and  private  bureaucracies:  A  transaction  cost 

economics  perspective.  Journai  of  Law,  Economics  and  Organization,  15, 
306-  42. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-37- 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


2003  -  2009  Sponsored  Research  Topics 


Acquisition  Management 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  BCA;  Contractor  vs.  Organic  Growth 

■  Defense  Industry  Consolidation 

■  EU-US  Defense  Industrial  Relationships 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to 
Shipyard  Planning  Processes 

■  Managing  the  Services  Supply  Chain 

■  MOSA  Contracting  Implications 

■  Portfolio  Optimization  via  KVA  +  RO 

■  Private  Military  Sector 

■  Software  Requirements  for  OA 

■  Spiral  Development 

■  Strategy  for  Defense  Acquisition  Research 

■  The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 

Contract  Management 

■  Commodity  Sourcing  Strategies 

■  Contracting  Government  Procurement  Functions 

■  Contractors  in  21®*-century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Model  for  Optimizing  Contingency  Contracting,  Planning  and  Execution 

■  Navy  Contract  Writing  Guide 

■  Past  Performance  in  Source  Selection 

■  Strategic  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  USAF  Energy  Savings  Performance  Contracts 

■  USAF  IT  Commodity  Council 

■  USMC  Contingency  Contracting 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Financial  Management 

■  Acquisitions  via  Leasing:  MRS  case 

■  Budget  Scoring 

■  Budgeting  for  Capabilities-based  Planning 

■  Capital  Budgeting  for  the  DoD 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Financing  DoD  Budget  via  PPPs 

■  Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition 
Budgeting  Reform 

■  PPPs  and  Government  Financing 

■  ROI  of  Information  Warfare  Systems 

■  Special  Termination  Liability  in  MDAPs 

■  Strategic  Sourcing 

■  Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 

Human  Resources 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 

■  Learning  Management  Systems 

■  Moral  Conduct  Waivers  and  First-tern  Attrition 

■  Retention 

■  The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 

■  Tuition  Assistance 

Logistics  Management 

Analysis  of  LAV  Depot  Maintenance 
Army  LOG  MOD 
ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
Diffusion/Variability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 

Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 

ACQUISITION  RESEARCH  PROGRAM 
GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 
NAVAL  POSTGRADUATE  SCHOOL 


■  Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

■  Optimizing  CIWS  Lifecycle  Support  (LCS) 

■  Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance 
Activity 

■  Pallet  Management  System 

■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

■  RFID  (6) 

■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 


A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our 
website:  www.acauisitionresearch.orq 


PERSCIWmm 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 

555  DYER  ROAD,  INGERSOLL  HALL 

MONTEREY,  CALIEORNIA  93943 


www.acquisitionresearch.org 


